fix(process-engine): survive optimistic-lock races on task, process, and context rows#151
Merged
Merged
Conversation
…and context rows
Under load the aggregator's backup flow kept dying with
VersionMismatchException ('Version in repository: 2, Task Version 1' /
'Current version are less than saved') from BackupDatabaseTask: several
actors hold independent in-memory copies of the same row — the POProcess
tick (every 2s), the executing task, ProcessTaskFailureHandler, and the
TaskExecutorService timeout watchdog — and the loser of a
milliseconds-wide race aborted its completion, leaving the db-scheduler
execution stuck picked until dead-execution revival. Externally that
surfaced as hung backups and HTTP 500s.
Three fixes:
- All three repositories now guard the UPDATE itself (WHERE version=?)
instead of a check-then-act compare, closing the silent lost-update
window, and bump the in-memory version only on success.
- Must-win writes recover instead of aborting: terminal COMPLETED/FAILED
saves go through TaskInstanceImpl.saveResolvingConflict (reload,
re-apply, save), and DataContext.apply re-applies its idempotent
mutation on a fresh copy after a conflict.
- TaskExecutionWrapper ran its callback twice on failure (catch plus
finally); it now runs exactly once and logs the error.
…after recovery Review findings on the race fix: - Process-level terminal writes (POProcess completion FAILED/COMPLETED, AsyncTaskWithPolling failure, the timeout watchdog) still called raw save() and could abort on the same version race the task writes were protected from. ProcessInstanceImpl gains the same saveResolvingConflict, used at all four sites; the pre-existing reload() is reused. - After conflict recovery both saveResolvingConflict variants and DataContext.apply now sync the caller's instance with the persisted state (contents, fields, version, clean dirty flag). Without that the caller kept a stale copy and the very next save() — for example the trailing process save in the POProcess completion handler — failed again despite the recovery having succeeded.
- FutureKey.equals no longer accepts String/UUID (equals-contract violation); terminate() now filters by getTaskId() explicitly (S2159) - re-interrupt the thread on InterruptedException in the watchdog and in terminate()'s wait loop (S2142) - terminate() returns Future<Void> instead of Future<?> (S1452) - TaskExecutionWrapper catches Exception, not Throwable (S1181) - ProcessTaskFailureHandler reuses a single Optional instance (S3655) - suppress S3011 (reflective access into db-scheduler internals) and S2160 (HashMap content equality is intentional) with justifications
kichasov
force-pushed
the
fix/process-engine-version-races
branch
from
July 23, 2026 19:02
7a8da29 to
36b4a12
Compare
An interrupt is the watchdog's normal shutdown signal: the completion callback in execute() cancels it with cancel(true) once the task finishes in time. The shared catch treated that interrupt as a timeout and marked the completed task and its process FAILED — and saveResolvingConflict made the bogus overwrite reliable instead of losing the version race. Split the catch: InterruptedException restores the interrupt flag and stops watching without touching state; only ExecutionException and TimeoutException cancel the worker and mark the task and process FAILED. The watchdog body moved to a package-private watchTask() so tests can drive both paths directly.
|
alsergs
approved these changes
Jul 24, 2026
kichasov
added a commit
to Netcracker/qubership-dbaas
that referenced
this pull request
Jul 24, 2026
…ck race fixes (#603) Picks up Netcracker/qubership-core-java-libs#151, merged to main and published as 1.5.5-20260724.091358-10: atomic version guards on the task, process, and context rows, conflict-recovering terminal saves, a single completion callback per task, and a watchdog that no longer fails a task that completed in time. AbstractDbaaSTask.updateState switches from put+save to context.apply so its status-description writes get the same conflict recovery; a plain save() aborted the whole task when the POProcess tick or the timeout watchdog wrote the same row first. Verified on verify/process-engine-race-fix and verify/pe-race-fix-nb-status against a throwaway snapshot of the same code: the DBaaS IT concurrent section passes and the VersionMismatchException signature is gone from the aggregator logs.
Logaka
pushed a commit
to Netcracker/qubership-dbaas
that referenced
this pull request
Jul 24, 2026
* chore(test-apps): bump dbaas client pins to refreshed mounted-secret snapshots Point the sample services at the mounted-secret libraries rebuilt on top of master: - spring: dbaas-client 9.1.3 -> 9.1.5-mounted-secret-SNAPSHOT - quarkus: dbaas-quarkus 10.1.1 -> 10.1.3-mounted-secret-SNAPSHOT - go: dbaas-base-client -> v3.5.4-0.20260701125619-f0a56365482c (feat/mounted-secret-provider merged with main), core-lib-go 3.11.0 -> 3.12.0, go directive 1.26.4 (required by deps). * docs(operator): normalize operator docs and comments to US English (#539) Apply the english-us-developer-style skill: normalize British spellings to American across the operator's user-facing docs, the Grafana dashboard description, metric Help strings, AGENTS.md, and code/dev/test comments; drop one filler word. Text-only, no logic changes. * test(sample-apps): added sample apps to test dbaas clients * test(sample-apps): add Phase C — direct M2M REST to the aggregator Add a third integration pass where all three sample clients (go/spring/quarkus) call the DBaaS aggregator directly over REST, authenticated with a Kubernetes projected service-account token (audience `dbaas`), instead of via the dbaas-agent sidecar. - helm (3 charts): dedicated ServiceAccount per app; a `dbaas`-audience projected token volume at /var/run/secrets/tokens/dbaas + KUBERNETES_M2M_ENABLED, gated on a new M2M_ENABLED value (default false — Pass A/B unchanged). M2M is activated without touching app sources, via helm env overrides: Spring flips DBAAS_RESTCLIENT_RESTTEMPLATE_BASIC_AUTH=false (selects the M2M OkHttp client); Quarkus clears the aggregator basic-auth creds (DbaasClientProducer picks the M2M client); Go gains API_DBAAS_ADDRESS (koanf api.dbaas.address). - aggregator: map the three sample-app ServiceAccounts to DB_CLIENT in service-account-roles.yaml. - deploy-sample-services.sh: optional third arg M2M_ENABLED, threaded to all charts. - workflow: grant the aggregator SA system:service-account-issuer-discovery (for OIDC token verification) + restart; add the Phase C deploy/wait/test block after Pass B (the phase-agnostic IT suite is reused unchanged). - docs: document Pass A/B/C + the M2M env in the three sample READMEs. * fix(spring-test-app): add dummy M2MManager for direct-M2M phase In Phase C the dbaas Spring client uses its M2M OkHttp flavor (basic-auth=false), whose dbaasRestClient bean autowires an M2MManager the app didn't provide, so it failed to start. Add a DummyM2MManager @bean (guarded by @ConditionalOnMissingBean). For direct dbaas calls the aggregator is authenticated with the pod's projected dbaas-audience token, not this manager's token, so the dummy is sufficient; the bean is unused in the basic-auth (Pass A/B) phases. * test(spring-test-app): add Tenant header to M2M dbaas client The dbaas Spring M2M flavor is an OkHttp client with no context-propagation interceptor (unlike the RestTemplate/Quarkus transports), so the Tenant header is dropped and tenant-scoped GetOrCreateDatabase fails the aggregator's classifier-vs-request tenant check (CORE-DBAAS-4054), which only runs for K8s-JWT (M2M) callers. Decorate the dbaasRestClient bean (via a BeanPostProcessor) to re-add the Tenant header from the current TenantContext. Temporary until OkHttp context propagation lands in the dbaas client library. * feat: CR conversion skill for DBaaS (#544) * feat: CR conversion skill for DBaaS * fix: removing the wrong file * fix: non chart sources now pass explicit values * chore: add APM agent support (#556) * chore(test-apps): drop redundant sample-app resource profiles (#554) Each sample-app chart shipped resource-profiles/{dev,dev-ha,prod,prod-nonha}.yaml, but all four files are identical and their five keys (CPU/MEMORY request/limit, REPLICAS) already match the values.yaml defaults. The deploy script overlaid resource-profiles/dev.yaml with `-f`, which changed nothing — `helm template` renders byte-for-byte identically with and without it. Remove the resource-profiles directories from all three charts and drop the `-f .../resource-profiles/dev.yaml` overlay from deploy-sample-services.sh. values.yaml already holds the resource and replica settings, so the rendered manifests are unchanged. These are throwaway CI test apps; the multi-tier profile convention adds no value here. * chore(ci): skip operator build for docs/dev-only commits (#553) The Go operator build ran on every push to main and every PR, including commits that touch only dbaas-operator/docs or dbaas-operator/dev — neither of which affects the built binary. Add paths-ignore for dbaas-operator/docs/** and dbaas-operator/dev/** to both the push and pull_request triggers. paths-ignore skips the run only when every changed file matches, so a mixed docs-plus-code commit still builds. Neither main nor feat/operator-dev requires Go Build as a status check, so a docs-only PR is not blocked by a skipped run. * chore(ci): align log artifact names, add sample-service diagnostics (#552) Both log artifacts read ambiguously (each looked like "just logs"). Rename them to say what each holds and which one to download: dbaas-pod-logs -> dbaas-pod-stdout (Fluent Bit stdout) dbaas-logs -> dbaas-k8s-diagnostics (kubectl describe/events/previous) sample-services-pod-logs -> sample-services-pod-stdout The sample-service workflow only exposed Fluent Bit stdout as an artifact; its kubectl diagnostics went to the job log only. Promote that dump to a downloadable sample-services-k8s-diagnostics artifact (same snapshot still folded into the console via ::group::), collected only on failure. That is the signal that matters when a pod never starts (ImagePull/CrashLoop/ OOMKilled), where the Fluent Bit stdout artifact is empty. Step names aligned to "Collect/Upload ... (tool)" across both workflows. * feat: skill for transforming DBaaS runtime provisioning (#546) * feat: CR conversion skill for DBaaS * fix: removing the wrong file * fix: non chart sources now pass explicit values * feat: first draft * fix: pushing fixes and review changes * fix: review changes * fix: review change * fix(ci): configure default S3 alias for the backup daemon (#593) Backup integration tests send `storageName: "default"` in every backup and restore request. The postgres backup daemon only resolves a storage name when S3 aliases are enabled, and rejects the request otherwise: storageName was provided but S3 aliases are not enabled The adapter turns that response into a panic and returns HTTP 500, so the aggregator marks the logical backup as RETRYABLE_FAIL and every BackupV3IT.Postgresql backup test fails. The pgskipper-operator chart enables aliases only when `backupDaemon.s3Aliases` is a non-empty list: an empty list leaves `s3AliasesUsed` unset in the CR, so the daemon starts without `S3_ALIASES_USED=true`. Declare one alias named `default` that points at the same MinIO tenant already configured under `s3Storage`. * fix(tests): read adapter credentials from the mounted secret (#594) * fix(ci): configure default S3 alias for the backup daemon Backup integration tests send `storageName: "default"` in every backup and restore request. The postgres backup daemon only resolves a storage name when S3 aliases are enabled, and rejects the request otherwise: storageName was provided but S3 aliases are not enabled The adapter turns that response into a panic and returns HTTP 500, so the aggregator marks the logical backup as RETRYABLE_FAIL and every BackupV3IT.Postgresql backup test fails. The pgskipper-operator chart enables aliases only when `backupDaemon.s3Aliases` is a non-empty list: an empty list leaves `s3AliasesUsed` unset in the CR, so the daemon starts without `S3_ALIASES_USED=true`. Declare one alias named `default` that points at the same MinIO tenant already configured under `s3Storage`. * fix(tests): read adapter credentials from the mounted secret The postgres adapter no longer publishes its basic-auth credentials through DBAAS_ADAPTER_API_USER and DBAAS_ADAPTER_API_PASSWORD. Since pgskipper-operator moved the adapter to mounted secrets, the credentials live in the `dbaas-adapter-credentials` secret, mounted at /var/run/secrets/postgresql/dbaas-adapter-credentials. MigrationHelper still fell back to `dbaas-aggregator-credentials.v1`, which is the mongo adapter's secret and does not exist in the postgres namespace. Every test that creates a database through the adapter API failed with: Key 'username' not found in secret 'dbaas-aggregator-credentials.v1' in namespace postgres Resolve the secret from the pod spec: find the volume mount whose path ends with `dbaas-adapter-credentials` and read the backing secret. Taking the name from the pod rather than hard-coding it keeps the helper working across adapters. The environment-variable lookup stays first, and the aggregator secret stays as the last fallback, so adapters on older versions keep working. * fix(aggregator): stop holding the repository mutex across Postgres writes (#597) DBAAS_REPOSITORIES_MUTEX is a single process-wide monitor that guards the in-memory H2 cache. Three write paths held it across a Postgres operation: DatabaseRegistryDbaasRepository.save (a Hibernate flush), and the two deleteById / delete methods (a Postgres delete). That created a deadlock spanning two layers. One thread would update a row, take its row lock, then block entering the monitor; the monitor holder, inside its own transaction, would flush an UPDATE that needed the same row and wait on the row lock. Neither side can detect it: Postgres sees the blocked thread's transaction as idle, and the JVM sees the monitor holder as RUNNABLE in a native socket read. The aggregator froze every mutex-guarded path until the pod restarted, surfacing as flaky OperatorIT and BlueGreenV1IT timeouts. Captured with thread dumps and pg_stat_activity in the integration-tests workflow. Do the Postgres write outside the monitor and hold the monitor only for the H2 mirror, matching PhysicalDatabaseDbaasRepository.delete and DatabaseDbaasRepository.deleteAll, which already follow this pattern. save now takes no monitor at all, like PhysicalDatabaseDbaasRepository.save, because its block is Postgres-only; the H2 mirror is refreshed asynchronously by the classifier table listener. * build(operator): generate the chart's CRD templates and verify them in CI (#590) * build(operator): generate the chart's CRD templates and verify them in CI The chart carries its own copy of every CRD so it can gate them on DBAAS_OPERATOR_ENABLED, and those copies were maintained by hand. They drifted. The chart shipped shortName "dbdp" for DatabaseAccessPolicy while the marker in api/v1 — unchanged since the resource was introduced — says "dbdap". Every documented `kubectl get dbdap` therefore failed against a chart-installed cluster. Three more templates carried stale descriptions, so `kubectl explain` answered from an older version of the API. Each template turned out to be exactly the generated CRD wrapped in the conditional and nothing else: all eight have their only two Helm expressions on the first and last line. That makes the copy mechanical, so add `make sync-helm-crds` to produce it, and depend on `manifests` so it can never be taken from a stale config/crd/bases. Add a verify-generated job that reruns manifests, generate and sync-helm-crds and fails on any diff. It catches the two mistakes that caused this — forgetting to regenerate, and forgetting to copy — and it would have caught the shortName divergence the day it appeared. Mark both CRD trees linguist-generated and record the target in AGENTS.md, so the next person edits the markers rather than the output. * fix(operator): delete orphan CRD templates and catch untracked drift sync-helm-crds only overwrote templates for CRDs that still exist, so removing a CRD from the API left its chart template behind and the chart kept installing it. Remove the generated crd-*.yaml files before the copy loop so the sync carries deletions too. The verify-generated job used git diff, which ignores untracked files: a brand-new CRD whose generated artifacts were never committed passed the check silently. Gate on git status --porcelain instead. * refactor(operator): make conditions the status contract, phase a summary (#585) * refactor(operator): make conditions the status contract, phase a summary The Kubernetes API conventions deprecate `phase` as a state model: "The pattern of using phase is deprecated. Newer API types should use conditions instead." The operator already models state correctly with Ready/Stalled conditions, but phase was still an input to control flow, and a closed OpenAPI enum on the status field made it an upgrade hazard. Stop reading phase in controller logic. Six predicates asked "is this terminal?" by comparing phase against Succeeded/InvalidConfiguration; they now derive the same answer from the conditions via isTerminal(). The DatabaseSecretClaim safety-net trigger mixed both mechanisms (root observedGeneration plus phase) and now uses isReadyForGeneration(), which reads the Ready condition's own observedGeneration — the field that exists for exactly this purpose. Drop the enum and the default from status.phase. A closed enum on a status field means that shipping a new phase value before the updated CRD reaches the cluster makes the API server reject the entire status write, conditions included, leaving the resource unobservable. Defaulting is equally wrong here: status is owned by the controller, which always sets phase alongside the conditions. Phase itself stays. It is the only way to show one readable column in `kubectl get`, because JSONPath can select but not branch. It is now documented as an observational summary, and a Ready column is added next to it so the authoritative signal is visible too. No behavior change: the predicates are equivalent, which the existing controller suite confirms, and new unit tests pin the equivalence. * fix(operator): make the terminal predicate generation-aware Conditions persist across reconciles, so after a spec change a reconcile that exits early (for example on a benign Secret create/update race) still carries the previous generation's Ready=True. isTerminal counted it as terminal, letting patchStatusOnExit stamp status.observedGeneration for a spec the controller never finished processing. The former phase-based predicate was immune: phase was reset to Processing at the start of every reconcile. Require the terminal condition's observedGeneration >= obj.Generation, pass the generation at all five shouldObserve sites, and add stale-Ready and stale-Stalled regression tests. * chore(operator): enforce balancing-rule name and blank values in the CRD (#587) The three balancing-rule CRDs carried no declarative validation at all, so every constraint was checked by the controller after the object already existed: a mistyped CR was admitted, reconciled, moved to InvalidConfiguration and left behind for someone to notice and delete. Two of those constraints are trivially expressible in the schema, and one of them is not fully covered by the controller either. Enforce the singleton name with CEL, matching what NamespaceBinding already does. A balancing rule is a singleton per namespace, and the name is fixed; rejecting a wrong name at admission gives the user the error in kubectl apply instead of a doomed object. Reject whitespace-only strings with Pattern=\S on type, dbType, name and physicalDatabaseId, and on the microservices and namespaces items. MinLength=1 does not cover this: " " has length 1 and passes. The controller's TrimSpace checks caught it, but only after admission. The controller-side checks stay as they are. They cover what the schema cannot express — uniqueness over the flattened (type x microservice) product, the operator-namespace check that depends on runtime config — and they keep working against a stale CRD. One existing spec created a second NamespaceBalancingRule under a non-singleton name to set up a rule-name collision. That shortcut no longer passes admission, so the spec now creates the conflicting singleton in another namespace, which is also the real scenario: aggregator rule names are global and each namespace has exactly one singleton. * chore(operator): align identifier casing with Go and API conventions (#588) * chore(operator): align identifier casing with Go and API conventions The CR types deliberately mirror the aggregator's Java DTOs, so the JSON names follow Java camelCase: tenantId, physicalDatabaseId, lastRequestId. That is a documented convention and it stays. One field broke it. Rename the status field trackingID to trackingId. The operator described the same value two ways: the aggregator client already used `json:"trackingId"`, while the CR status used trackingID. The wire and the CR now agree, and the field matches the four other Id-suffixed names in the API. Nothing reads the field by its JSON name in tests, so the change is confined to the tag, the regenerated CRD, and the docs — including a kubectl jsonpath example that would otherwise have silently returned nothing. Fix the Go identifiers that violate the initialism rule: TenantId, DbName and DbType become TenantID, DBName and DBType, and three stray Id struct fields become ID. JSON tags are untouched, so this changes no API surface. Enable revive's var-naming rule to keep it that way. It immediately found the three Id fields that manual review had missed. The rule covers Go identifiers only; the camelCase JSON tags are unaffected. One case is deliberately left alone: tmf.Response.Id in the aggregator mock is a field of an external library type. * chore(operator): rename DbType in tests merged from feat/operator-dev The balancing-rule validation tests landed in the base branch with the pre-rename field name and missed the DBType sweep. * docs(operator): record why the required label cannot be validated by CEL (#589) The comment on DatabaseSecretClaim said CEL validation of metadata.labels "is not supported by controller-gen". That is not where the limit lives, and the wording sends the next reader down a dead end: controller-gen generates such a rule without complaint, and the CRD only fails when the API server compiles it. A root-level XValidation rule sees a trimmed metadata — name and generateName only. Referencing self.metadata.labels makes the API server reject the entire CRD with "undefined field 'labels'"; self.metadata.namespace fails the same way, which also rules out a CEL check that spec.classifier.namespace matches metadata.namespace. Both were verified against a live API server. Record the real reason and both consequences, so the controller-side pre-flight checks read as a deliberate choice rather than an oversight. The regenerated CRD also picks up description drift that had accumulated between config/crd/bases and the hand-maintained Helm template for this resource. * fix: adding the finalizer permission for secret claim (#586) * fix: adding the finalizer permission for secret claim * fix: aligning with the cluster role changes * feat: enforcing owner ref RBAC in kind envs * docs(operator): add databasesecretclaims/finalizers to the permission reference The RBAC rule landed without its row in the permission table, which claims to explain why each permission is needed. --------- Co-authored-by: Roman Kichasov <roman.kichasov.qubership@gmail.com> * fix(ci): pin kind v0.32.0 for kubeadm v1beta4 config helm/kind-action@v1.14.0 defaults to kind v0.31.0, which only generates kubeadm v1beta3 config where extraArgs is a string map. The admission-plugin patch added in #586 uses the v1beta4 list form, so kubeadm failed to parse the merged config and cluster creation broke both integration workflows. kind v0.32.0 is the first release with the v1beta4 template. * fix(ci): force snapshot updates when building sample apps The sample apps pin throwaway -SNAPSHOT builds of the dbaas clients, and the Maven cache restored by setup-java carries snapshot metadata. With the default daily update policy, a same-day publish-then-test cycle reuses the cached snapshot instead of the one just published. -U makes Maven re-resolve snapshots on every build. * build(test-apps): consume spring client from main, quarkus from the rebuilt throwaway Spring: the mounted-secret support merged to main (dbaas-client 9.1.7-SNAPSHOT, includes the DatabasePool cache fixes), so drop the 9.1.5-mounted-secret throwaway pin. Quarkus: the throwaway republished at 10.1.5-mounted-secret-SNAPSHOT after merging the deduplicated feature branch; its client classes now come from dbaas-client-base. * build(test-apps): consume the Go dbaas client from main (#600) The Go sample app was pinned to a throwaway feature-branch build of qubership-core-lib-go-dbaas-base-client. The mounted-secret support has merged to main (Netcracker/qubership-core-lib-go-dbaas-base-client#70), so switch to the main pseudo-version (no release tag covers the merge yet) and pick up the transitive qubership-core-lib-go 3.13.1 bump. * fix(ci): keep per-pass Surefire reports in sample-service tests (#601) All three sample-service passes (mount proof, REST fallback, direct M2M) run `mvn test` in the same module, so each pass overwrote the previous pass's target/surefire-reports. The final gate and the uploaded artifact only ever saw the last pass, and the nightly job stayed green while pass A had been failing 12/12 on main since July 10. Give every pass its own reports directory. Surefire 3.x exposes no user property for reportsDirectory, so route it through a Maven property in the test module's POM, the same way testFailureIgnore is already wired. The gate now checks each pass's report set separately and treats a missing set as a failure, so a pass that never wrote reports cannot pass silently. The diagnostics trigger reads the per-pass directories too; the artifact glob already covers subdirectories. * build(test-apps): consume the quarkus dbaas client from main The mounted-secret support merged to main (Netcracker/qubership-core-java-libs#117), so drop the throwaway 10.1.5-mounted-secret pin — all three sample apps now consume the main snapshots of their clients. * feat(operator): report NamespaceBinding state in status (#599) * feat(operator): report NamespaceBinding state in status NamespaceBinding was the only CR without a status: registration was visible only as a finalizer, a blocked deletion only as a Warning event that expires after an hour, and a checker failure only in operator logs. Give it the shared OperatorStatus and write it from the owning instance: - Ready=True/BindingRegistered (phase Succeeded) once the finalizer is in place; Ready=False/BindingBlocked (phase Processing) while deletion is deferred; BackingOff/OwnershipCheckError when the blocking-resource check fails. Foreign instances keep writing nothing, so an unclaimed binding (operatorNamespace typo) is recognizable by its empty status. - BlockingResourceChecker now returns the blocking kind names instead of a bare bool; the Ready message and the BindingBlocked event say what exactly is in the way. - patchStatusOnExit tolerates NotFound: releasing the last finalizer can let the deletion complete before the deferred status patch runs. - LastRequestID is assigned inside the defer — the finalizer patch refreshes the object from the API response and would wipe it. - NamespaceBinding joins dbaas_resource_phase/condition/generation_lag for owned bindings; dbaas_namespace_binding_state stays as is. Verified in kind: registered → Succeeded/True; EDB + delete → Processing with 'deletion deferred: ExternalDatabase resources still present in the namespace' and observedGeneration held back; EDB removed → binding released. * docs(operator): stop claiming NamespaceBinding has no status The usage-examples section still said the finalizer was the single indicator and argued a status field would add no value — contradicting the status reference added in this PR and hiding the main diagnostic for BindingBlocked and OwnershipCheckError. Point it at status.phase/Ready, keep the finalizer as a supplementary signal, and show how to read the blocking-kinds message when a deletion hangs. Add the missing Status Reference ToC entry. * fix(operator): stop amplifying workload churn into binding writes Every workload watch event re-enqueued the binding, and after the status change each such reconcile issued a status patch — LastRequestID alone made the patch non-empty every time. During the concurrent IT section that turned the binding into an API-write amplifier. Filter the six workload watches down to create/delete events (spec and status updates cannot change whether blocking resources exist), and skip the exit patch when the status did not change, stamping LastRequestID only on a real write. * fix(operator): address binding status review findings - A foreign finalizer can keep the binding alive after the protection finalizer is removed, leaving a BindingBlocked condition that names resources that no longer exist. The release path now rewrites the conditions to Ready=False/BindingReleased; when nothing else holds the object the deferred patch hits NotFound and is skipped as before. - The six workload watches on NamespaceBinding fired on every update, so each binding status write re-enqueued all workloads in the namespace. Ownership only changes with create, delete, or a spec change — filter the watches with GenerationChangedPredicate. - lastRequestId documented as the ID of the most recent reconcile that wrote the status: a steady-state no-op reconcile keeps the previous value, which the field's contract now says instead of contradicting. * fix(operator): survive a failed release status patch Removing the protection finalizer and writing BindingReleased happen in one reconcile; when that status patch failed transiently, the retry hit the finalizer-already-removed branch, changed nothing, and the stale BindingBlocked survived forever on a binding held by another finalizer. The branch now re-asserts the released conditions through the shared markBindingReleased helper — a no-op write when the status already says released. Regression test simulates the failing first patch. Also document the BindingReleased state in the status reference and the CRD godoc, which previously pointed only at BindingBlocked and OwnershipCheckError for a hung deletion. * docs(operator): list all NamespaceBinding reasons in the conditions godoc The shared conditions godoc named only BindingRegistered and BindingBlocked; BindingReleased and OwnershipCheckError were documented in the howto table but missing from the API godoc and the generated CRD descriptions. * fix(aggregator): pin process-engine 1.5.5-SNAPSHOT with optimistic-lock race fixes (#603) Picks up Netcracker/qubership-core-java-libs#151, merged to main and published as 1.5.5-20260724.091358-10: atomic version guards on the task, process, and context rows, conflict-recovering terminal saves, a single completion callback per task, and a watchdog that no longer fails a task that completed in time. AbstractDbaaSTask.updateState switches from put+save to context.apply so its status-description writes get the same conflict recovery; a plain save() aborted the whole task when the POProcess tick or the timeout watchdog wrote the same row first. Verified on verify/process-engine-race-fix and verify/pe-race-fix-nb-status against a throwaway snapshot of the same code: the DBaaS IT concurrent section passes and the VersionMismatchException signature is gone from the aggregator logs. * chore: updating operator comments (#575) * chore: updating operator comments * chore(operator): regenerate CRDs after comment updates The comment updates touched the shared Conditions field doc, which feeds the generated CRD descriptions. Run make manifests and make sync-helm-crds so the verify-generated job matches what is committed. * docs(operator): clarify writeSecret convergence sequence * docs(operator): correct status and polling comments * docs(operator): address review feedback --------- Co-authored-by: Roman Kichasov <roman.kichasov.qubership@gmail.com> --------- Co-authored-by: Abhimanyu Roy <51357041+TheHollowMonarch@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Why
The dbaas-aggregator's backup flow keeps failing under load with
VersionMismatchExceptionfromBackupDatabaseTask(four DBaaS IT runs bitten so far; diagnostics in Netcracker/qubership-dbaas runs 29988579829 and 30026600214):Several actors hold independent in-memory copies of the same
pe_task_instance/po_context/ process rows — the POProcess tick (every 2 s), the executing task,ProcessTaskFailureHandler, and theTaskExecutorServicetimeout watchdog. Whoever loses a milliseconds-wide race aborts its completion; the db-scheduler execution then stays picked until dead-execution revival, and externally that is a hung backup and an HTTP 500.Two defects compound it: the repositories used a check-then-act version compare (SELECT, compare in Java, unguarded UPDATE), which under true concurrency also silently loses one of two writes; and
TaskExecutionWrapperran its completion callback twice on failure.What
WHERE ... AND version=?, affected-row check) and bump the in-memory version only after success. Exception contract unchanged.COMPLETED/FAILEDsaves go through newTaskInstanceImpl.saveResolvingConflict(reapply)— on a conflict the row is reloaded, the idempotent mutation re-applied, and the fresh copy saved (used inAbstractProcessTask,ProcessTaskFailureHandler,AsyncTaskWithPolling, the timeout watchdog).DataContext.applydoes the same for context mutations, which covers task start/end times and state descriptions.TaskExecutionWrapperruns its callback exactly once and logs the swallowed error.A completion that today crashes with a stuck execution would, with slightly different timing, have succeeded anyway — the recovery path persists exactly that outcome, so no new interleavings are introduced.
How to verify
mvn -pl core-process-orchestrator test— 37 tests green, including the newVersionConflictTest: stale-copy failures for all three repositories (winning write survives, no corruption), terminal-save recovery via reload, contextapplyrecovery preserving the concurrent writer's data, and single-callback wrapper behavior.After merge, the consumer side (Netcracker/qubership-dbaas) needs a
process-enginebump (currently pinned to 1.5.2) plus switchingAbstractDbaaSTask.updateStatetocontext.apply(...)so its status-description writes get the same recovery — I will follow up there and re-run the DBaaS IT that reproduces the race.